Papers with noise-contrastive estimation
Pre-Training Transformers as Energy-Based Cloze Models (2020.emnlp-main)
Copied to clipboard
| Challenge: | elucidates close connection between cloze modeling and representation learning over text. |
| Approach: | They propose an energy-based cloze model for representation learning over text . they assign a scalar energy score to each input token indicating how likely it is given context . |
| Outcome: | The proposed model performs better than masked language models and faster than cloze models. |
Documents Representation via Generalized Coupled Tensor Chain with the Rotation Group constraint (2021.findings-acl)
Copied to clipboard
| Challenge: | despite the diversity of linguistic structures, vector embedding models lack order-preserving properties . current methods for learning linguistic structure can be expensive and time-consuming . |
| Approach: | They propose a method for embedding documents and words in rotation group . they capture word order and higher-order word interactions . |
| Outcome: | The proposed model achieves the best results in document classification benchmarks. |
Improving Unsupervised Relation Extraction by Augmenting Diverse Sentence Pairs (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies on relation representation learning focus on contrastive learning strategies, but these studies overlook important aspects. |
| Approach: | They propose to use within-sentence pairs augmentation and cross-sentent pairs extraction to increase diversity of positive pairs and strengthen the discriminative power of contrastive learning. |
| Outcome: | The proposed task increases diversity of positive pairs and strengthens discriminative power . it overcomes limitations of traditional Relation Extraction tasks, which require manual annotations . |